{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. 动态渲染页面的爬取技术\n",
    "\n",
    "**JavaScript**动态渲染的页面不止Ajax这一种。\n",
    "\n",
    "比如中国青年网 [http://news.youth.cn/gn/](http://news.youth.cn/gn/),\n",
    "它的分页部分是由 JavaScript生成的，并非原始 HTML 代码，\n",
    "这其中并不包含 Ajax请求。 比如 ECharts 的官方实例\n",
    "[http://echarts.baidu.com/demo.html#bar-negative](http://echarts.baidu.com/demo.html#bar-negative)，\n",
    "其图形都是经过 JavaScript计算之后生成的。\n",
    "有淘宝这种页面，它使是 Ajax 获取的数据，但是其Ajax接口含有很多加密参数，\n",
    "我们难以直接找出其规律，也很难直接分析Ajax来抓取\n",
    "\n",
    "为了解决这些问题，我们可以直接使用模拟浏览器运行的方式来实现，\n",
    "这样就可以做到在浏览器中是什么样，抓取的源码就是什么样\n",
    "\n",
    "这样我们就不用再去管网页内部的 JavaScript用了什么算法渲染页面，不用管网页后台的**Ajax**接口到底有哪些参数。\n",
    "\n",
    "Python 提供了许多模拟浏览器运行的库，如 Selenium、Splash、PyV8,Ghost等。\n",
    "本章中，我们就来介绍一下**Selenium**和**Splash**的用法。\n",
    "\n",
    "## 6.1 Selenium 的使用\n",
    "\n",
    "    Selenium是一个自动化测试工具，利用它可以驱动浏览器执行特定的动作，如点击、下拉等操作，\n",
    "同时还可以获取浏览器当前呈现的页面的源代码，做到可见即可爬。对于一些JavaScript 动态渲染的 页面来说，此种抓取方式非常有效\n",
    "\n",
    "### 本章的项目的浏览器的WebDriver使用Microsoft Edge()实现,其他的也参考相关文献和百度。\n",
    "\n",
    "Selenium支持非常多的浏览器，如Chrome、Firefox、Edge等，还有Android、Black.Berry等手机端的浏览器。 另外，也支持无界面浏览器PhantomJS。\n",
    "\n",
    "我们可以用get()方法来请求网页，参数传入链接URL即可。比如，这里用get()方法访问淘宝，然后打印出源代码\n",
    "\n",
    "## 6.1.1 淘宝前端页面的简单爬取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#1. 淘宝的简单爬取\n",
    "from selenium import webdriver\n",
    "from bs4 import BeautifulSoup\n",
    "browser = webdriver.Edge()\n",
    "browser.get(\"https://www.taobao.com\")\n",
    "t = browser.page_source\n",
    "print(t)\n",
    "r = BeautifulSoup(t).prettify()\n",
    "print(r)\n",
    "browser.close()#关闭浏览驱动器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1.2 节点查找\n",
    "Selenium 可以驱动浏览器完成各种操作，\n",
    "比如填充表单、模拟点击等。\n",
    "比如，我们想要完成向某个输入框输入文字的操作，\n",
    "总需要知道这个输入框在哪里吧？\n",
    "而**Selenium**提供了一系列查找节点的方法，我们可以用这些方法来获取想要的节点，\n",
    "以便下一步执行一些动作或者提取信息。\n",
    "\n",
    "比如，想要从淘宝页面中提取搜索框这个节点，首先要观察它的源代码\n",
    "![](taobaosearch.jpg)\n",
    "![](taobao2.jpg)\n",
    "可以发现，它的id是q, name也是如此外，还有许多其他属性，此时我们就可以用多种方式获取它了。\n",
    "比如，find_element_by_name()是根据name值获取，find_element_by_id()是根据id获取。\n",
    "\n",
    "另外，还有根据XPath、css选择器等获取的方式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "browser = webdriver.Edge()\n",
    "browser.get('https://www.taobao.com')\n",
    "inputvalue = browser.find_element_by_id('q')\n",
    "print('标准获取：\\n',inputvalue)\n",
    "inputvalue2 = browser.find_element_by_xpath(\"//*[@id='q']\")\n",
    "print('XPath获取：',inputvalue2)\n",
    "browser.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外，selenium支持下面的模式获取单个节点"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "find_element_by_id\n",
    "find_element_by_name\n",
    "find_element_by_xpath\n",
    "find_element_by_link_text\n",
    "find_element_by_partial_link_text\n",
    "find_element_by_tag_name\n",
    "find_element_by_class_name\n",
    "find_element_by_css_selector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外，Selenium还提供了通用方法 find_element()，它需要传入两个参数：\n",
    "查找方式By和值。\n",
    "\n",
    "实际上，它就是find_element_by_id()这种方法的通用函数版本\n",
    "\n",
    "find_element(By.ID,'q')#查找id=q的相关数据\n",
    "\n",
    "### 多节点\n",
    "\n",
    "类比单节点，多节点只是添加使用了find_elements()而已，不再赘述\n",
    "\n",
    "### 节点交互（重要）\n",
    "\n",
    "Selenium可以驱动浏览器来执行一些操作，\n",
    "也就是说可以让浏览器模拟执行一些动作。\n",
    "比较常见的用法有：输入文字时用 send_keys()方法，\n",
    "清空文字时用clear()方法，点击按钮时用click()方法。\n",
    "示例如下：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "browser = webdriver.Edge()\n",
    "browser.get('https://www.taobao.com')\n",
    "inputvalue = browser.find_element_by_id('q')#查找锁定id=q的节点\n",
    "inputvalue.send_keys('iPad')#发送iPad的表单\n",
    "time.sleep(1)#等待1s\n",
    "inputvalue.clear()\n",
    "inputvalue.send_keys('iPhone')\n",
    "button = browser.find_element_by_class_name('btn-search')\n",
    "#按钮找btn-search节点\n",
    "button.click()#点击按钮"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里首先驱动浏览器打开淘宝，然后用find_element_by_id()方法获取输入框，\n",
    "然后用 send_keys()方法输入iPhone文字，\n",
    "等待一秒后用 clear()方法清空输入框，\n",
    "再次调用send_keys()方法输入iPad文字，\n",
    "之后再用find_element_by_class_name()方法获取搜索按钮，\n",
    "最后调用click()方法完成搜索动作。\n",
    "\n",
    "通过上面的方法，我们就完成了一些常见节点的动作操作，\n",
    "更多的操作可以参见官方文档的交互动作介绍：\n",
    "\n",
    "[http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement](http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement)\n",
    "\n",
    "### 动作链\n",
    "\n",
    "在上面的实例中，一些交互动作都是针对某个节点执行的。\n",
    "比如，对于输入框，我们就调用它的输入文字和清空文字方法；\n",
    "对于按钮，就调用它的点击方法。\n",
    "\n",
    "其实，还有另外一些操作，它们没有特定的执行对象，\n",
    "比如鼠标拖曳、键盘按键等，这些动作用另一种方式来执行，那就是动作链。\n",
    "\n",
    "比如，现在实现一个节点的拖曳操作，将某个节点从一处拖曳到另外一处，可以这样实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver import ActionChains\n",
    "browser = webdriver.Edge()\n",
    "url = 'http://www.runoob.com/try/try.php?filename=jqueryui-api-droppable'\n",
    "browser.get(url)\n",
    "browser.switch_to.frame('iframeResult')\n",
    "source = browser.find_element_by_css_selector('#draggable')\n",
    "target= browser.find_element_by_css_selector('#droppable')\n",
    "actions = ActionChains(browser)#定义对象动作\n",
    "actions.drag_and_drop(source, target)\n",
    "actions.perform()#执行动作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，打开网页中的一个拖曳实例，\n",
    "然后依次选中要拖曳的节点和拖曳到的目标节点，\n",
    "接着声明**ActionChains**对象并将其赋值为**actions**变量，\n",
    "然后通过调用actions变量的 drag_and_drop()方法，\n",
    "再调用perform()方法执行动作自动\n",
    "\n",
    "更多的动作链操作可以参考官方文档：\n",
    "\n",
    "[http://selenium-python.readthedocs.io/api.html#moduleselenium.webdriver.common.action_chains](http://selenium-python.readthedocs.io/api.html#moduleselenium.webdriver.common.action_chains)\n",
    "\n",
    "### 执行JavaScript\n",
    "\n",
    "对于某些操作，Selenium的API并没有提供。\n",
    "比如，下拉进度条，它可以直接模拟运行JavaScript的语言实现\n",
    "\n",
    "此时使用execute_script()方法即可实现，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "browser= webdriver.Edge()\n",
    "browser.get('https://www.baidu.com/s?wd=Python')\n",
    "browser.execute_script('window.scrollTo(0,document.body.scrollHeight)')\n",
    "browser.execute_script('alert(\"To Bottom\")')\n",
    "#执行脚本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里就利用execute_script()方法将进度条下拉到最底部，然后弹出alert提示框。\n",
    "\n",
    "所以说有了这个方法，基本上API没有提供的所有功能都可以用**执行JavaScript**的方式来实现了。\n",
    "\n",
    "### 获取节点信息\n",
    "\n",
    "既然Selenium已经提供了选择节点的方法，返回的是WebElement类型，那么它也有相关\n",
    "的方法和属性来直接提取节点信息，如属性、文本等。这样的话，我们就可以不用通过解析源代码来\n",
    "提取信息了，非常方便。\n",
    "\n",
    "我们可以使用get_attribute()方法来获取节点的属性，但是其前提是先选中这个节点，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver import ActionChains\n",
    "browser = webdriver.Edge()\n",
    "url='https://www.zhihu.com/explore'\n",
    "browser.get(url)\n",
    "logo= browser.find_element_by_id('zh-top-link-logo')\n",
    "print(logo)\n",
    "print(logo.get_attribute('class'))#获取匹配属性\n",
    "input = browser.find_element_by_class_name('zu-top-add-question')\n",
    "print(input.text)#获取文本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 异常处理\n",
    "\n",
    "**上面的代码出现了异常，现在进行自定义处理。**\n",
    "\n",
    "在使用Selenium的过程中，难免会遇到一些异常，\n",
    "例如超时、节点未找到等错误，一旦出现此类问题，程序便不会继续运行了。\n",
    "这里我们可以使用try-except语句来捕获各种异常。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.common.exceptions \\\n",
    "    import TimeoutException, NoSuchElementException\n",
    "browser = webdriver.Edge()\n",
    "try:\n",
    "    browser.get('https://www.baidu.com')\n",
    "except TimeoutException:\n",
    "    print(' Time Out')\n",
    "try:\n",
    "    browser.find_element_by_id('hello')\n",
    "except NoSuchElementException:\n",
    "    print ('No Element')\n",
    "finally:\n",
    "    browser.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于更多的异常类，可以参考官方文档：\n",
    "\n",
    "[http://selenium-python.readthedocs.io/api.html#module-selenium.common.exceptions](http://selenium-python.readthedocs.io/api.html#module-selenium.common.exceptions)\n",
    "\n",
    "### 选项卡管理\n",
    "\n",
    "在访问网页的时候，会开启一个个选项卡。在Selenium中，我们也可以对选项卡进行操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from selenium import webdriver\n",
    "browser = webdriver.Edge()\n",
    "browser.get('https://www.baidu.com')\n",
    "browser.execute_script('window.open()')#JS语句：开一个选项卡\n",
    "print(browser.window_handles)\n",
    "time.sleep(1)\n",
    "browser.switch_to.window(browser.window_handles[1])\n",
    "browser.get('http://www.hao123.com')\n",
    "time.sleep(1)\n",
    "browser.switch_to.window(browser.window_handles[0])\n",
    "browser.get('http://www.python.org')\n",
    "time.sleep(1)\n",
    "browser.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意：版本有改动，switch_to.window()是现行版本\n",
    "\n",
    "### Cookies 处理(增删改查)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "browser.add_cookie({\"name\":\"name\",\"wd\":\"Python\",\"location\":\"Germany\"})\n",
    "print(browser.get_cookies())\n",
    "browser.delete_all_cookies()\n",
    "print(browser.get_cookies())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模拟浏览器前进后退"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "browser = webdriver.Edge()\n",
    "browser.get('http://www.baidu.com')\n",
    "time.sleep(0.5)\n",
    "browser.get('https://www.taobao.com')\n",
    "time.sleep(0.5)\n",
    "browser.get('http://www.hao123.com')\n",
    "time.sleep(0.5)\n",
    "browser.back()#回退\n",
    "time.sleep(1)\n",
    "browser.back()#回退\n",
    "time.sleep(1)\n",
    "browser.forward()#前进"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们连续访问3个页面，然后调用back()和forward方法进行交互，回退以及前进\n",
    "\n",
    "### 延时等待\n",
    "\n",
    "在Selenium 中，get()方法会在网页框架加载结束后结束执行，此时如果获取page_source\n",
    "可能并不是浏览器完全加载完成的页面，如果某些页面有额外的Ajax请求，我们在网页代码中也不一\n",
    "定能成功获取到。所以，这里需要延时等待一定时间，确保节点已经加载出来。\n",
    "这里等待的方式有两种：一种是隐式等待，一种是显式等待。\n",
    "\n",
    "#### 1.隐式等待\n",
    "\n",
    "使用隐式等待执行测试的时候，如果Selenium没有在DOM中找到节点，将继续等待，超出设\n",
    "定时间后，则抛出找不到节点的异常。换句话说，当查找节点而节点并没有立即出现的时候，隐式等\n",
    "待将等待一段时间再查找DOM，默认的时间是0。\n",
    "\n",
    "**这里我们用implicitly_wait()方法实现了隐式等待。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.common.exceptions \\\n",
    "    import TimeoutException, NoSuchElementException\n",
    "browser = webdriver.Edge()\n",
    "try:\n",
    "    browser.implicitly_wait(10)#等待最大值10s\n",
    "    browser.get(\"https://www.zhihu.com/explore\")\n",
    "    input = browser.find_element_by_class_name('zu-top-add-question')\n",
    "    print(input)\n",
    "except NoSuchElementException as n:\n",
    "    print(\"找不到节点\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.显式等待\n",
    "\n",
    "隐式等待的效果其实并没有那么好，因为我们只规定了一个固定时间，而页面的加载时间会受到\n",
    "网络条件的影响。\n",
    "这里还有一种更合适的显式等待方法，它指定要查找的节点，\n",
    "然后指定一个最长等待时间。如果在规定时间内加载这个节点，\n",
    "就返回查找的节点； 如果到了规定时间依然没有加载出该节点，\n",
    "则抛出超时异常。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as DC\n",
    "\n",
    "browser = webdriver.Edge()\n",
    "browser.get('https://www.taobao.com')\n",
    "wait = WebDriverWait(browser,10)\n",
    "input = wait.until(DC.presence_of_element_located((By.ID,'q')))\n",
    "button = wait.until(DC.element_to_be_clickable((By.CSS_SELECTOR,'.btn-search')))\n",
    "print(input,button)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里首先引人WebDriverWait 这个对象，指定最长等待时间，然后调用它的until()方法，\n",
    "传入要等待条件expected_conditions。比如，这里传入了presence_of_element_located 这个条件，代表节点出现的意思，其参数是节点的定位元组，也就是ID为q的节点搜索框。\n",
    "\n",
    "这样可以做到的效果就是，在10秒内如果ID 为q的节点成功加载出来，就返回该\n",
    "节点；如果超过10秒还没有加载出来，就抛出异常。\n",
    "\n",
    "关于等待条件，其实还有很多，比如判断标题内容，判断某个节点内是否出现了某文字等。\n",
    "\n",
    "参考下面图表\n",
    "![](seletime.jpg)\n",
    "![](seletime2.jpg)\n",
    "\n",
    "关于更多等待条件的参数及用法，可以参考官方文档：\n",
    "[http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.support.expected_conditions](http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.support.expected_conditions)\n",
    "\n",
    "### Frame切换\n",
    "\n",
    "我们知道网页中有一种节点叫作iframe,也就是子Frame，相当于页面的子页面，\n",
    "它的结构和外部网页的结构完全一致。Selenium 打开页面后，它默认是在父级Frame 里面操作，而此时如果页面中\n",
    "还有子Frame，它是不能获取到子Frame里面的节点的。这时就需要使用\n",
    "switch_to.frame()方法来切换Frame 。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "from selenium.common.exceptions import NoSuchElementException\n",
    "url = 'http://www.runoob.com/try/try.php?filename=jqueryui-api-droppable'\n",
    "browser = webdriver.Edge()\n",
    "browser.get(url)\n",
    "browser.switch_to.frame('iframeResult')\n",
    "try:\n",
    "    logo = browser.find_element_by_class_name('logo')\n",
    "except NoSuchElementException as e:\n",
    "    print('No logos')\n",
    "browser.switch_to.parent_frame()\n",
    "logo = browser.find_element_by_class_name('logo')\n",
    "print(logo)\n",
    "print(logo.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 这里还是以前面演示动作链操作的网页为实例\n",
    "\n",
    "首先通过switch_to.frame()方法换到子Frame\n",
    "里面，然后尝试获取父级Frame里的logo节点（这是不能找到的），\n",
    "如果找不到的话，就会抛出NoSuchElementException异常，\n",
    "异常被捕捉之后，就会输出NO LOGO。接下来，重新切换回父级Frame,\n",
    "然后再次重新获取节点，发现此时可以成功获取了。\n",
    "\n",
    "### 弹出的对话框处理"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "switch_to_alert() 　　#定位弹出对话\n",
    "text() 　　   #获取对话框文本值\n",
    "accept()      #相当于点击\"确认\"\n",
    "dismiss()     #相当于点击\"取消\"\n",
    "send_keys()   # 输入值，这个alert和confirm没有输入对话框，所以这里就不能用了，所以这里只能使用在prompt这里。\n",
    "\n",
    "#基本操作流程：\n",
    "import time\n",
    "from selenium import webdriver\n",
    "driver = webdriver.Chrome()\n",
    "driver.maximize_window()\n",
    "driver.get(URL)\n",
    "#获取alert对话框的按钮,点击按钮,弹出alert对话框\n",
    "driver.find_element_by_xpath('/html/body/div/input[2]').click()\n",
    "#获取alert对话框\n",
    "alert = driver.switch_to_alert()\n",
    "#添加等待时间\n",
    "time.sleep(2)\n",
    "#获取警告对话框的内容\n",
    "print (alert.text)  #打印警告对话框内容\n",
    "alert.accept()   #alert对话框属于警告对话框，我们这里只能接受弹窗\n",
    "#添加等待时间\n",
    "time.sleep(2)\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 是否有界面（Google Chrome webdriver required）\n",
    "无界面：Headless模式"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "chromeopt = webdriver.ChromeOptions()\n",
    "chromeopt.add_argument('--headless')\n",
    "# browser = webdriver.Chrome()\n",
    "browser = webdriver.Chrome(chrome_options=chromeopt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 项目实践1：打卡系统自动化运维\n",
    "\n",
    "1.进入打卡系统实现页面切换，并进入要求的位置输入相应的值\n",
    "\n",
    "2.操作时注意观察开发者工具，对节点等进行分析！\n",
    "\n",
    "3.请使用GUI界面工具，实现界面友好交互！\n",
    "\n",
    "4.注意数据的存储！\n",
    "\n",
    "项目文件：ScriptSign.py\n",
    "\n",
    "## 项目实践2：淘宝爬取\n",
    "\n",
    "要求除了不用界面设计，其他同上\n",
    "\n",
    "1.注意数据的存储，放在Mongodb里面！\n",
    "\n",
    "项目文件：TaobaoSpyder.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 Splash 的使用\n",
    "(本项目需要在Docker虚拟环境下配置)个人使用的是Toolbox虚拟环境配置\n",
    "这里使用了阿里云的镜像加速的设计，实现快速安装访问\n",
    "\n",
    "使用终端或者git Bash进入docker后输入下面指令，\n",
    "安装代码："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "docker run -p 8050:8050 scrapinghub/splash"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](dockersplash.jpg)\n",
    "\n",
    "随后进入下面的页面，注意，在浏览器输入终端显示的ip+:8050(端口号).如：192.168.122.102:8050等\n",
    "看到下面的图片\n",
    "\n",
    "![](splashbrowser.JPG)\n",
    "Splash是一个JavaScript渲染服务,\n",
    "是一个带有HTTPAPI的轻量级浏览器，\n",
    "同时它对接了Python中的Twisted和QT库。\n",
    "利用它，我们同样可以实现动态页面的抓取。\n",
    "\n",
    "拥有功能：\n",
    "\n",
    "异步方式处理多个网页渲染过程；获取渲染后的页面的源代码或截图；\n",
    "\n",
    "通过关闭图片渲染或者使用Adblock规则来加快页面渲染速度；\n",
    "\n",
    "可执行特定的JavaScript脚本；\n",
    "\n",
    "可通过Lua脚本来控制页面渲染过程；\n",
    "\n",
    "获取渲染的详细过程并通过HAR(HTTP Archive)格式呈现。\n",
    "\n",
    "如图所示，网页右侧呈现的是一个渲染示例。\n",
    "可以看到，上方有一个输入框，\n",
    "默认是 http://google.com, 这里换成百度测试一下，\n",
    "将内容更改为 https://www.baidu.com\n",
    "\n",
    "然后点击Render me按钮开始渲染，看到结果：\n",
    "\n",
    "![](baidusplash.jpg)\n",
    "\n",
    "网页的返回结果呈现了渲染截图、HAR加载统计数据、网页的源代码。\n",
    "\n",
    "通过HAR的结果可以看到，Splash执行了整个网页的渲染过程，\n",
    "包括css、JavaScript 的加载等过程，\n",
    "呈现的页面和我们在浏览器中得到的结果完全一致。\n",
    "那么，这个过程由什么来控制呢？\n",
    "重新返回首页，可以看到实际上是有一段脚本，内容如下"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  assert(splash:go(args.url))\n",
    "  assert(splash:wait(0.5))\n",
    "  return {\n",
    "    html = splash:html(),\n",
    "    png = splash:png(),\n",
    "    har = splash:har(),\n",
    "  }\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个脚本实际上是用Lua语言写的脚本。\n",
    "\n",
    "即使不懂这个语言的语法，但从脚本的表面意思，\n",
    "我们也可以大致了解到它首先调用go()方法去加载页面，然后调用wait()方法等待了一定时间，\n",
    "最后返回了页面的代码、截图和HAR信息。\n",
    "到这里，我们大体了解了Splash是通过Lua脚本来控制了页面的加载过程的，\n",
    "加载过程完全模拟 浏览器，最后可返回各种格式的结果，如网页源码和截图等。\n",
    "接下来，我们就来了解Lua脚本的写法以及相关API的用法。\n",
    "\n",
    "### Lua 脚本简介\n",
    "\n",
    "Splash可以通过Lua脚本执行一系列渲染操作，\n",
    "这样我们就可以用Splash来模拟类似Chrome、PhantomJS的操作了。\n",
    "\n",
    "### 1. 入口、返回值"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash,args)\n",
    "    splash:go(\"http://www.baidu.com\")\n",
    "    splash:wait(0.5)\n",
    "    local title = splash:evaljs(\"document.title\")\n",
    "    return {title = title}\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "代码粘贴到刚才打开的url界面下的代码编辑区域，\n",
    "然后点击 Render me！按钮来测试一下。看到它返回了网页的标题\n",
    "\n",
    "这里我们通过evaljs()方法传入JavaScript脚本，\n",
    "而document.title的执行结果就是返回网页标题，\n",
    "执行完毕后将其赋值给一个title变量，随后将其返回。\n",
    "\n",
    "注意：我们在这里定义的方法名称叫作 main()。\n",
    "这个名称必须是固定的，Splash会默认调用这个方法。\n",
    "\n",
    "该方法的返回值既可以是字典形式，也可以是字符串形式，\n",
    "最后都会转化为 Splash HTTP Response 例如："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash)\n",
    "    return {hello=\"world\"}\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的是字典式，下面的是字符串式"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash)\n",
    "    return \"hello\"\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.异步处理\n",
    "\n",
    "Splash 支持异步处理，但是这里并没有显式指明回调方法，\n",
    "其回调的跳转是在Splash内部完成的\n",
    "示例："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash,args)\n",
    "    local example_urls = {\"www.baidu.com\",\"www.hao123.com\",\"www.zhihu.com\"}\n",
    "    local urls = args.urls or example_urls\n",
    "    local results = {}\n",
    "    for index,url in ipairs(urls) do\n",
    "        local ok,reason = splash:go(\"http://\" .. url)\n",
    "        if ok then\n",
    "            splash:wait(2)\n",
    "            results[url] = splash:png()\n",
    "        end\n",
    "    end\n",
    "    return results\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到的结果：\n",
    "![](urlsplashes.jpg)\n",
    "\n",
    "在脚本内调用的 wait()方法类似于Python中的 time.sleep()，\n",
    "其参数为等待的秒数。\n",
    "当 Splash 执行到此方法时，它会转而去处理其他任务，\n",
    "然后在指定的时间过后再回来继续处理。\n",
    "\n",
    "这里值得注意的是，Lua 脚本中的字符串拼接和 Python 不同，\n",
    "它使用的是 .. 操作符，而不是+。\n",
    "\n",
    "如果有必要，可以简单了解一下Lua脚本的语法，详见\n",
    "[http://www.runoob.com/lua/lua-basic-syntax.html](http://www.runoob.com/lua/lua-basic-syntax.html)\n",
    "\n",
    "另外，这里做了加载时的异常检测。\n",
    "\n",
    "go()方法会返回加载页面的结果状态，如果页面出现4xx或5xx状态码，\n",
    "ok变量就为空，就不会返回加载后的图片。\n",
    "\n",
    "### 3.Splash 对象属性分析\n",
    "\n",
    "main()方法的第一个参数是splash，这个对象非常重要，它类似于\n",
    "Selenium中的WebDriver对象，我们可以调用它的些属性和方法来控制加载过程\n",
    "\n",
    "args\n",
    "该属性可以获取加载时配置的参数，比如 URL ，如果为 GET 请求，\n",
    "它还可以获取 GET 请求参数；\n",
    "如果为 POST 请求，\n",
    "它可以获取表单提交的数据Splash支持使用第2个参数直接作为args"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "    local url = args.url\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里第二个参数**args**就相当于 splash.args 属性.\n",
    "以上代码等价于："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash)\n",
    "    local url = splash.args.url\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.json_enabled 属性\n",
    "\n",
    "这个属性是Splash的JavaScript 执行开关，\n",
    "可以将其配置为 true/false 来控制是否执行JavaScript\n",
    "代码，默认为 true 例如，这里禁止执行JavaScript"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash,args)\n",
    "    splash:go(\"https://www.baidu.com\")\n",
    "    splash.js_enabled = false\n",
    "    local title = splash:evaljs(\"document.title\")\n",
    "    return {title=title}\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "弹出报错：\n",
    "重新调用了evaljs()方法执行JavaScript 代码时，\n",
    "此时运行结果就会抛出异常："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "{\n",
    "    \"error\": 400,\n",
    "    \"type\": \"ScriptError\",\n",
    "    \"description\": \"Error happened while executing Lua script\",\n",
    "    \"info\": {\n",
    "        \"type\": \"JS_ERROR\",\n",
    "        \"js_error_message\": null,\n",
    "        \"message\": \"[string \\\"function main(splash,args)\\r...\\\"]:4: unknown JS error: None\",\n",
    "        \"splash_method\": \"evaljs\",\n",
    "        \"source\": \"[string \\\"function main(splash,args)\\r...\\\"]\",\n",
    "        \"line_number\": 4,\n",
    "        \"error\": \"unknown JS error: None\"\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般来说，不用设置此属性，默认开启即可\n",
    "\n",
    "### 5.resource_timeout\n",
    "\n",
    "此属性可以设置加载的超时时间，单位是秒;如果设置为nil\n",
    "(类似 Python 中的 None)，代表不检测超时\n",
    "\n",
    "如下面的代码表示超时限制为0.1s"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "splash.resource_timeout = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此属性适合在网页加载速度较慢的情况下设置，如果超过了某个时间无响应，则直接抛出异常\n",
    "忽略即可\n",
    "\n",
    "### 6.images_enabled 属性\n",
    "\n",
    "此属性可以设置图片是否加载，默认情况下是加载的禁用该属性后，\n",
    "可以节省网络流量并提高网页加载速度.\n",
    "但是需要注意的是，禁用图片加载可能会影响 JavaScript 渲染。因为禁用图片之后，\n",
    "它的外层DOM节点的高度会受影响，进而影响 DOM 节点的位置。因此，如果 JavaScript 对图片节点有操作的话，其执行就会受到影响\n",
    "\n",
    "另外值得注意的是，Splash使用了缓存,如果一开始加载出来了网页图片，\n",
    "然后禁用了图片加载，再重新加载页面，之前加载好的图片可能还会显示出来，\n",
    "这时直接重启 Splash 即可"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "    splash.images_enabled = false\n",
    "    assert(splash:go(\"https://www.jd.com\"))\n",
    "    return {png=splash:png()}\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结果如下，显示的界面也是没有图片渲染的。\n",
    "![](Jdsplash.JPG)\n",
    "\n",
    "### 7.plugins_enabled 属性\n",
    "\n",
    "此属性可以控制浏览器插件（如 Flash 插件）是否开启.默认情况下，此属性是false,表示不开启。\n",
    "可以使用如下代码控制其开启和关闭：\n",
    "\n",
    "**splash.plugins_enabled = true/false**\n",
    "\n",
    "### 8.scroll_position\n",
    "\n",
    "通过设置此属性，我们可以控制页面上下或左右滚动.这是一个比较常用的属性"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "    assert(splash:go('https://www.taobao.com'))\n",
    "    splash.scroll_position = {y=400}\n",
    "    return { png=splash: png() }\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上的代码实现了控制页面向下滚动400像素值的效果\n",
    "如果要让页面左右滚动，可以传入x参数,如取x=100像素把上面第二行数据改成：\n",
    "\n",
    "**splash.scroll_position = {x=100,y=400}**\n",
    "\n",
    "## Splash对象操作\n",
    "\n",
    "### 9.go()方法\n",
    "\n",
    "该方法用来请求某个链接，而且它可以模拟GET和POST请求，\n",
    "同时支持传入请求头、表单等数据，其用法如下："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "ok, reason = splash:go{url, baseurl=nil, headers=nil, http_method=\"GET\", body=nil , formdata=nil}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "url：请求的url网址\n",
    "\n",
    "baseurl：可选参数 ，默认为空，表示资源加载相对路径\n",
    "\n",
    "headers：可选参数，默认为空，表示请求头\n",
    "\n",
    "http_method：可选参数，默认为GET，同时支持 POST\n",
    "\n",
    "body：可选参数，默认为空，发POST请求时的表单数据，\n",
    "使用的Content-type为application/json\n",
    "\n",
    "formdata：可选参数，默认为空，POST的时候的表单数据，使用的 Content-type为\n",
    "application/x-www-form-urlencoded\n",
    "\n",
    "该方法的返回结果是结果ok和原因reason组合，\n",
    "如果ok为空，代表网页加载出现了错误，此时\n",
    "reason变量中包含了错误的原因，否则证明页面加载成功，示例如下："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "    local ok,reason=splash:go{\"http://httpbin.org/post\",http_method=\"POST\",body=\"name=Germey\"}\n",
    "    if ok then\n",
    "        return splash:html()\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到下面结果\n",
    "![](postsplash.jpg)\n",
    "可以看到，我们成功实现了 POST 请求并发送了表单数据\n",
    "\n",
    "### 10.wait()方法\n",
    "\n",
    "此方法可以控制页面的等待时间，使用方法如下："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "ok, reason = splash:wait{time, cancel_on_redirect=false, cancel_on_error=true}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "time等待的秒数\n",
    "\n",
    "cancel_on_redirect ：可选参数，默认为false，表示如果发生了重定向就停止等待，\n",
    "井返回重定向结果\n",
    "\n",
    "cancel_on_error 可选参数，默认为 false，表示如果发生了加载错误，就停止等待\n",
    "\n",
    "返回结果同样是结果 ok 和原因 reason 的组合"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash)\n",
    "    splash:go(\"https://www.taobao.com\")\n",
    "    splash:wait(2)\n",
    "    return {html=splash:html()}\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.jsfunc() 方法\n",
    "此方法可以直接调用JavaScript定义的方法，但是所调用的方法需要用双中括号包围，\n",
    "这相当于实现了JavaScript方法到 Lua 脚本的转换"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "    local get_div_count = splash:jsfunc([[\n",
    "    function() {\n",
    "        var body = document.body;\n",
    "        var divs = body.getElementsByTagName('div');\n",
    "        return divs.length;\n",
    "        }\n",
    "    ]])\n",
    "    splash:go(\"https://www.baidu.com\")\n",
    "    return (\"There are %s DIVs\"):format(get_div_count())\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回结果： Splash Response: \"There are 53 DIVs\"\n",
    "首先，我们声明了一个JavaScript定义的方法，\n",
    "然后在页面加载成功后调用了此方法计算出了页面中div节点的个数\n",
    "\n",
    "关于 JavaScript -> Lua 脚本的更多转换细节，可以参考官方文档：\n",
    "\n",
    "[https://splash.readthedocs.io/en/stable/scripting-ref.html#splash-jsfunc](https://splash.readthedocs.io/en/stable/scripting-ref.html#splash-jsfunc)\n",
    "\n",
    "### 12.evaljs()方法\n",
    "\n",
    "此方法可以执行JavaScript代码并返回最后一条JavaScript语句的返回结果\n",
    "比如，可以用下面的代码来获取页面标题："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "local title = splash:evaljs(\"document.title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 13.runjs()方法\n",
    "\n",
    "此方法可以执行JavaScript代码，它与evaljs()的功能类似，\n",
    "但是更偏向于执行某些动作或某些方法"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash,args)\n",
    "    splash:go(\"https://www.baidu.com\")\n",
    "    splash:runjs(\"foo = function(){ return 'bar'}\")\n",
    "    local result = splash:evaljs(\"foo()\")\n",
    "    return result\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回结果：Splash Response: \"bar\"\n",
    "\n",
    "这里我们用runjs()先声明了一个JavaScript定义的方法，然后通过evaljs()来调用得到的结果\n",
    "\n",
    "### 14.autoload()\n",
    "\n",
    "此方法可以设置每个页面访问时自动加载的对象"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "ok,reason = splash:autoload{source_or_url,source=nil,url=nil}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参数说明：\n",
    "source_or_url: JavaScript代码或者JavaScript 库链接\n",
    "source: JavaScript 代码\n",
    "url: JavaScript 库链接\n",
    "\n",
    "但是此方法只负责加载JavaScript代码或库，不执行任何操作.\n",
    "如果要执行操作，可以调用 evaljs()和runjs()的方法"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  splash:autoload([[\n",
    "    function get_document_title(){\n",
    "    return document.title\n",
    "  }\n",
    "    ]])\n",
    "  splash:go(\"https:www.baidu.com\")\n",
    "  return splash:evaljs(\"get_document_title()\")\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到结果：Splash Response: \"百度一下，你就知道\"\n",
    "\n",
    "这里我们调用autoload()方法声明了一个JavaScript方法，\n",
    "然后通过evaljs()方法来执行此JavaScript()该方法。\n",
    "\n",
    "### 15.call_later()\n",
    "\n",
    "此方法可以通过设置定时任务和延迟时间来实现任务延时执行，\n",
    "并且可以在执行前通过cancel()方法重新执行定时任务。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  local snapshots={}\n",
    "  local timer=splash:call_later(function()\n",
    "      snapshots[\"a\"]=splash:png()\n",
    "      splash:wait(1.0)\n",
    "      snapshots[\"b\"]=splash:png()\n",
    "  end,0.2)\n",
    "  splash:go(\"https://www.taobao.com\")\n",
    "  splash:wait(3.0)\n",
    "  return snapshots\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回结果：\n",
    "![](calllatersplash.jpg)\n",
    "\n",
    "这里我们设置了一个定时任务，0.2秒的时候获取网页截图，然后等待1秒，1.2秒时再次获取网页截图，\n",
    "访问的页面是淘宝，最后将截图结果返回。\n",
    "\n",
    "### 16.http_get()\n",
    "\n",
    "此方法可以模拟发送HTTP的GET请求，使用方法如下:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "response=splash:http_get{url,headers=nil,follow_redirects=true}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "url:请求URL\n",
    "headers:可选参数，默认为空，请求头\n",
    "follow_redirects：可选参数，表示是否启动自动重定向，默认为true。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  local treat=require(\"treat\")\n",
    "  local response=splash:http_get(\"http://httpbin.org/get\")\n",
    "     return {\n",
    "          html=treat.as_string(response.body),\n",
    "          url=response.url,\n",
    "          status=response.status\n",
    "  }\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到下面的结果："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Splash Response: Object\n",
    "html: String (length 395)download\n",
    "{\n",
    "  \"args\": {},\n",
    "  \"headers\": {\n",
    "    \"Accept-Encoding\": \"gzip, deflate\",\n",
    "    \"Accept-Language\": \"en,*\",\n",
    "    \"Host\": \"httpbin.org\",\n",
    "    \"User-Agent\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1\",\n",
    "    \"X-Amzn-Trace-Id\": \"Root=1-5f326352-c6e533d61bcecb889f60d28a\"\n",
    "  },\n",
    "  \"origin\": \"112.17.236.3\",\n",
    "  \"url\": \"http://httpbin.org/get\"\n",
    "}\n",
    "\n",
    "status: 200\n",
    "url: \"http://httpbin.org/get\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 17.http_post()\n",
    "\n",
    "和http_get()方法类似，此方法用来模拟发送POST请求，不过多了一个参数body"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "response=splash:http_post{url,headers=nil,follow_redirects=true,body=nil}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "url:请求URL。\n",
    "headers:可选参数，默认为空，请求头\n",
    "follow_redirects::可选参数，表示是否启动自动重定向，默认为true。\n",
    "body:可选参数，即表单数据，默认为空。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  local treat=require(\"treat\")\n",
    "  local json=require(\"json\")\n",
    "  local response=splash:http_post{\"http://httpbin.org/post\",\n",
    "    body=json.encode({name=\"Germey\"}),\n",
    "    headers={[\"content-type\"]=\"application/json\"}\n",
    "    }\n",
    "  return {\n",
    "    html=treat.as_string(response.body),\n",
    "    url=response.url,\n",
    "    status=response.status\n",
    "  }\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结果："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "plash Response: Object\n",
    "html: String (length 573)download\n",
    "{\n",
    "  \"args\": {},\n",
    "  \"data\": \"{\\\"name\\\": \\\"Germey\\\"}\",\n",
    "  \"files\": {},\n",
    "  \"form\": {},\n",
    "  \"headers\": {\n",
    "    \"Accept-Encoding\": \"gzip, deflate\",\n",
    "    \"Accept-Language\": \"en,*\",\n",
    "    \"Content-Length\": \"18\",\n",
    "    \"Content-Type\": \"application/json\",\n",
    "    \"Host\": \"httpbin.org\",\n",
    "    \"User-Agent\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1\",\n",
    "    \"X-Amzn-Trace-Id\": \"Root=1-5f326683-d1513a4a4de93daecd4e815a\"\n",
    "  },\n",
    "  \"json\": {\n",
    "    \"name\": \"Germey\"\n",
    "  },\n",
    "  \"origin\": \"112.17.236.3\",\n",
    "  \"url\": \"http://httpbin.org/post\"\n",
    "}\n",
    "\n",
    "status: 200\n",
    "url: \"http://httpbin.org/post\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们成功模拟提交了POST请求并发生了表单数据\n",
    "\n",
    "### 18.set_content()\n",
    "\n",
    "此方法用来设置页面的内容，示例如下:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash)\n",
    "    assert(splash:set_content(\"<html><body><h1>hello splash</h1></body></html>\"))\n",
    "    return splash:png()\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到结果：![](hellosplash.jpg)\n",
    "\n",
    "### 19.html()\n",
    "\n",
    "此方法用来获取网页的源代码，它是非常简单又常用的方法。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  splash:go(\"https://httpbin.org/get\")\n",
    "  return splash:html()\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 20.png()\n",
    "\n",
    "此方法用来获取PNG格式的网页截图"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  splash:go(\"https://www.taobao.com\")\n",
    "  return splash:png()\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 21.jpeg()\n",
    "\n",
    "此方法用来获取JPEG格式的网页截图,上面的代码改成下面即可"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "return splash:jpeg()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 22.har() url()\n",
    "\n",
    "har()方法用来获取页面加载过程描述，有具体相关的图片，url方法可以获取当前正在访问的URL。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  splash:go(\"https://www.taobao.com\")\n",
    "  return splash:har()   --return splash:url()\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 23.get_cookies()\n",
    "\n",
    "此方法可以获取当前页面的Cookies"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  splash:go(\"https://www.baidu.com\")\n",
    "  return splash:get_cookies()\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 24.add_cookie() clear_cookies()\n",
    "\n",
    "add_cookie()方法可以为当前页面添加Cookie\n",
    "\n",
    "clear_cookies() 可以清除所有的cookies splash:clear_cookies()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "cookies=splash:add_cookie{name,value,path=nil,expires=nil,httpOnly=nil,secure=nil}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 25.get_viewport_size() set_viewport_size()\n",
    "\n",
    "可以获取/设置当前浏览器页面的大小，即宽高\n",
    "\n",
    "### 26.set_viewport_full()\n",
    "\n",
    "可以设置浏览器全屏显示\n",
    "\n",
    "### 27.set_user_agent()\n",
    "\n",
    "此方法可以设置浏览器的User-Agent"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  splash:set_user_agent('Splash')\n",
    "  splash:go(\"https://www.baidu.com\")\n",
    "  return splash:html()\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回结果：\n",
    "\n",
    "**Splash Response: \"<html><head>\\n\\t<script>\\n\\t\\tlocation.replace(location.href.replace(\\\"https://\\\",\\\"http://\\\"));\\n\\t</script></head></html>\"**\n",
    "\n",
    "### 28.set_custom_headers()\n",
    "\n",
    "此方法可以设置请求头，添加方法：\n",
    "splash:set_custom_headers({[\"user-agent\"]=\"Splash\",[\"Site\"]=\"Splash\"})\n",
    "\n",
    "\n",
    "### 29.select()\n",
    "该方法可以选中符合条件的第一个节点，如果有多个节点符合条件，则只会返回一个，其参数是CSS选择器"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  splash:go(\"https://www.baidu.com/\")\n",
    "  input=splash:select(\"#kw\")\n",
    "  input:send_text(\"Splash\")\n",
    "  splash:wait(3)\n",
    "  return splash:png()\n",
    "\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结果：\n",
    "![](selectsplash.jpg)\n",
    "首先访问了百度，然后选中了搜索框，随后调用了send_text()方法填写了文本，然后返回网页截图。\n",
    "\n",
    "### 30.select_all()\n",
    "\n",
    "此方法可以选中所有符合条件的节点，其参数是CSS选择器"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  local treat=require('treat')\n",
    "  assert(splash:go(\"http://quotes.toscrape.com/\"))\n",
    "  assert(splash:wait(0.5))\n",
    "  local texts=splash:select_all('.quote .text')\n",
    "  local results={}\n",
    "  for index,text in ipairs(texts) do\n",
    "    results[index]=text.node.innerHTML\n",
    "  end\n",
    "  return treat.as_array(results)\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里通过CSS选择器选中了节点的正文内容，随后遍历了所有节点，将其中的文本获取下来\n",
    "\n",
    "运行截图：\n",
    "\n",
    "![](selectssplashes.jpg)\n",
    "\n",
    "### 31.mouse_click()\n",
    "\n",
    "此方法可以模拟鼠标点击操作，传入的参数为坐标值x和y。此外，也可以直接选中某个节点，然后调用此方法"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "function main(splash, args)\n",
    "  splash:go(\"https://www.baidu.com/\")\n",
    "  input=splash:select(\"#kw\")\n",
    "  input:send_text('Splash')\n",
    "  submit=splash:select('#su')\n",
    "  submit:mouse_click()\n",
    "  splash:wait(3)\n",
    "  return splash:png()\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们首先选中页面的输入框，输入文本，然后选中提交按钮，调用了mouse_click()方法提交查询，\n",
    "然后页面等待三秒，返回截图。\n",
    "\n",
    "前面介绍了splash常用API操作，还有API在这不再一一介绍，更加详细和权威的说明可\n",
    "以参见官方文档\n",
    "[https://splash.readthedocs.io/en/stable/scripting-ref.html](https://splash.readthedocs.io/en/stable/scripting-ref.html)\n",
    "页面介绍了splash对象的所有API操作\n",
    "\n",
    "另外，还有针对页面元素的 API 操作，链接为:\n",
    "\n",
    "[https://splash.readthedocs.io/en/stable/scripting-element-object.html](https://splash.readthedocs.io/en/stable/scripting-element-object.html)\n",
    "\n",
    "## Splash API Python接口调用\n",
    "\n",
    "前面说明了Splash Lua脚本的用法，但这些脚本是在 Splash 页面中测试运行的，\n",
    "如何才能利用Splash 渲染页面呢？\n",
    "怎样才能和 Python 程序结合使用并抓取 JavaScript 渲染的页面呢？用API调用！\n",
    "\n",
    "Splash给我们提供了HTTPAPI接口，我们只需要请求这些接口并传递相应的参数即可\n",
    "\n",
    "### 1. render.html\n",
    "\n",
    "此接口用于获取JavaScript渲染的页面的HTML代码，接口地址就是Splash的运行地址加此接口的名称\n",
    "\n",
    "如果用Python实现的话，如下所示:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "url='http://192.168.99.102:8050/render.html?url=https://www.baidu.com'\n",
    "response=requests.get(url)\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这样就可以成功输出百度页面渲染后的源代码，此接口还可以指定其他参数\n",
    "\n",
    "### 2.render.png\n",
    "\n",
    "此接口可以获取网页截图，其参数比render.html多了几个，比如通过width和height来控制宽高，它返回的是PNG格式的图片二进制数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "url='http://192.168.99.102:8050/render.png?url=https://www.jd.com&wait=5&wait=1000&height=700'\n",
    "response=requests.get(url)\n",
    "t = response.content\n",
    "print(t)\n",
    "# with open('taobaotest.png','wb') as f:\n",
    "#     f.write(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.render.jpeg\n",
    "此接口和render.png不过它返回的是JPEG格式的图片二进制数据\n",
    "\n",
    "另外，此接口比render.png多了个参数quality，用来设置图片质量。\n",
    "\n",
    "### 4.render.har\n",
    "\n",
    "此接口用于获取页面加载的HAR数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"log\": {\"version\": \"1.2\", \"creator\": {\"name\": \"Splash\", \"version\": \"3.4.1\"}, \"browser\": {\"name\": \"QWebKit\", \"version\": \"602.1\", \"comment\": \"PyQt 5.13.1, Qt 5.13.1\"}, \"entries\": [{\"_splash_processing_state\": \"finished\", \"startedDateTime\": \"2020-08-12T02:59:22.953159Z\", \"request\": {\"method\": \"GET\", \"url\": \"https://www.baidu.com/\", \"httpVersion\": \"HTTP/1.1\", \"cookies\": [], \"queryString\": [], \"headers\": [{\"name\": \"User-Agent\", \"value\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1\"}, {\"name\": \"Accept\", \"value\": \"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\"}], \"headersSize\": 189, \"bodySize\": -1}, \"response\": {\"bodySize\": 287459, \"httpVersion\": \"HTTP/1.1\", \"cookies\": [{\"name\": \"BAIDUID\", \"value\": \"593438C106AB1CB06A8D07285821B5D9:FG=1\", \"path\": \"/\", \"domain\": \".baidu.com\", \"expires\": \"2088-08-30T06:13:30Z\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BIDUPSID\", \"value\": \"593438C106AB1CB06A8D07285821B5D9\", \"path\": \"/\", \"domain\": \".baidu.com\", \"expires\": \"2088-08-30T06:13:30Z\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"PSTM\", \"value\": \"1597201163\", \"path\": \"/\", \"domain\": \".baidu.com\", \"expires\": \"2088-08-30T06:13:30Z\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BAIDUID\", \"value\": \"593438C106AB1CB0C0FECD1609F4848C:FG=1\", \"path\": \"/\", \"domain\": \".baidu.com\", \"expires\": \"2021-08-12T02:59:23Z\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BDSVRTM\", \"value\": \"14\", \"path\": \"/\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BD_HOME\", \"value\": \"1\", \"path\": \"/\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"H_PS_PSSID\", \"value\": \"32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\", \"path\": \"/\", \"domain\": \".baidu.com\", \"httpOnly\": false, \"secure\": false}], \"headers\": [{\"name\": \"Bdpagetype\", \"value\": \"1\"}, {\"name\": \"Bdqid\", \"value\": \"0xcee0a98b000391a1\"}, {\"name\": \"Cache-Control\", \"value\": \"private\"}, {\"name\": \"Connection\", \"value\": \"keep-alive\"}, {\"name\": \"Content-Encoding\", \"value\": \"gzip\"}, {\"name\": \"Content-Type\", \"value\": \"text/html;charset=utf-8\"}, {\"name\": \"Date\", \"value\": \"Wed, 12 Aug 2020 02:59:23 GMT\"}, {\"name\": \"Expires\", \"value\": \"Wed, 12 Aug 2020 02:59:23 GMT\"}, {\"name\": \"P3p\", \"value\": \"CP=\\\" OTI DSP COR IVA OUR IND COM \\\", CP=\\\" OTI DSP COR IVA OUR IND COM \\\"\"}, {\"name\": \"Server\", \"value\": \"BWS/1.1\"}, {\"name\": \"Set-Cookie\", \"value\": \"BAIDUID=593438C106AB1CB06A8D07285821B5D9:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com\\nBIDUPSID=593438C106AB1CB06A8D07285821B5D9; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com\\nPSTM=1597201163; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com\\nBAIDUID=593438C106AB1CB0C0FECD1609F4848C:FG=1; max-age=31536000; expires=Thu, 12-Aug-21 02:59:23 GMT; domain=.baidu.com; path=/; version=1; comment=bd\\nBDSVRTM=14; path=/\\nBD_HOME=1; path=/\\nH_PS_PSSID=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498; path=/; domain=.baidu.com\"}, {\"name\": \"Strict-Transport-Security\", \"value\": \"max-age=172800\"}, {\"name\": \"Traceid\", \"value\": \"1597201163093767629814907101181062123937\"}, {\"name\": \"X-Ua-Compatible\", \"value\": \"IE=Edge,chrome=1\"}, {\"name\": \"Transfer-Encoding\", \"value\": \"chunked\"}], \"content\": {\"size\": 0, \"mimeType\": \"text/html;charset=utf-8\"}, \"headersSize\": 1135, \"ok\": true, \"url\": \"https://www.baidu.com/\", \"status\": 200, \"statusText\": \"OK\", \"redirectURL\": \"\"}, \"cache\": {}, \"timings\": {\"blocked\": -1, \"dns\": -1, \"connect\": -1, \"ssl\": -1, \"send\": 1, \"wait\": 114, \"receive\": 328}, \"time\": 443, \"pageref\": \"1\"}, {\"_splash_processing_state\": \"finished\", \"startedDateTime\": \"2020-08-12T02:59:23.389336Z\", \"request\": {\"method\": \"GET\", \"url\": \"https://sp1.baidu.com/-L-Xsjip0QIZ8tyhnq/v.gif?logactid=1234567890&showTab=10000&opType=showpv&mod=superman%3Alib&submod=index&superver=supernewplus&glogid=3167470754&type=2011&pid=315&isLogin=0&version=PCHome&terminal=PC&qid=3167470909&sid=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498&super_frm=&from_login=&from_reg=&query=&curcard=2&curcardtab=&_r=0.944085872012462\", \"httpVersion\": \"HTTP/1.1\", \"cookies\": [{\"name\": \"BIDUPSID\", \"value\": \"593438C106AB1CB06A8D07285821B5D9\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"PSTM\", \"value\": \"1597201163\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BAIDUID\", \"value\": \"593438C106AB1CB0C0FECD1609F4848C:FG=1\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"H_PS_PSSID\", \"value\": \"32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}], \"queryString\": [{\"name\": \"logactid\", \"value\": \"1234567890\"}, {\"name\": \"showTab\", \"value\": \"10000\"}, {\"name\": \"opType\", \"value\": \"showpv\"}, {\"name\": \"mod\", \"value\": \"superman%3Alib\"}, {\"name\": \"submod\", \"value\": \"index\"}, {\"name\": \"superver\", \"value\": \"supernewplus\"}, {\"name\": \"glogid\", \"value\": \"3167470754\"}, {\"name\": \"type\", \"value\": \"2011\"}, {\"name\": \"pid\", \"value\": \"315\"}, {\"name\": \"isLogin\", \"value\": \"0\"}, {\"name\": \"version\", \"value\": \"PCHome\"}, {\"name\": \"terminal\", \"value\": \"PC\"}, {\"name\": \"qid\", \"value\": \"3167470909\"}, {\"name\": \"sid\", \"value\": \"32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\"}, {\"name\": \"super_frm\", \"value\": \"\"}, {\"name\": \"from_login\", \"value\": \"\"}, {\"name\": \"from_reg\", \"value\": \"\"}, {\"name\": \"query\", \"value\": \"\"}, {\"name\": \"curcard\", \"value\": \"2\"}, {\"name\": \"curcardtab\", \"value\": \"\"}, {\"name\": \"_r\", \"value\": \"0.944085872012462\"}], \"headers\": [{\"name\": \"User-Agent\", \"value\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1\"}, {\"name\": \"Referer\", \"value\": \"https://www.baidu.com/\"}, {\"name\": \"Accept\", \"value\": \"*/*\"}, {\"name\": \"Cookie\", \"value\": \"BIDUPSID=593438C106AB1CB06A8D07285821B5D9; PSTM=1597201163; BAIDUID=593438C106AB1CB0C0FECD1609F4848C:FG=1; H_PS_PSSID=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\"}], \"headersSize\": 348, \"bodySize\": -1}, \"response\": {\"bodySize\": 0, \"httpVersion\": \"HTTP/1.1\", \"cookies\": [], \"headers\": [{\"name\": \"Access-Control-Allow-Credentials\", \"value\": \"true\"}, {\"name\": \"Access-Control-Allow-Methods\", \"value\": \"GET, POST, OPTIONS\"}, {\"name\": \"Access-Control-Allow-Origin\", \"value\": \"*\"}, {\"name\": \"Cache-Control\", \"value\": \"no-cache\"}, {\"name\": \"Connection\", \"value\": \"keep-alive\"}, {\"name\": \"Content-Length\", \"value\": \"0\"}, {\"name\": \"Content-Type\", \"value\": \"image/gif\"}, {\"name\": \"Date\", \"value\": \"Wed, 12 Aug 2020 02:59:23 GMT\"}, {\"name\": \"Pragma\", \"value\": \"no-cache\"}, {\"name\": \"Server\", \"value\": \"nginx/1.8.0\"}, {\"name\": \"Tracecode\", \"value\": \"35635177880338144522081210\"}], \"content\": {\"size\": 0, \"mimeType\": \"image/gif\"}, \"headersSize\": 330, \"ok\": true, \"url\": \"https://sp1.baidu.com/-L-Xsjip0QIZ8tyhnq/v.gif?logactid=1234567890&showTab=10000&opType=showpv&mod=superman%3Alib&submod=index&superver=supernewplus&glogid=3167470754&type=2011&pid=315&isLogin=0&version=PCHome&terminal=PC&qid=3167470909&sid=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498&super_frm=&from_login=&from_reg=&query=&curcard=2&curcardtab=&_r=0.944085872012462\", \"status\": 200, \"statusText\": \"OK\", \"redirectURL\": \"\"}, \"cache\": {}, \"timings\": {\"blocked\": -1, \"dns\": -1, \"connect\": -1, \"ssl\": -1, \"send\": 0, \"wait\": 112, \"receive\": 0}, \"time\": 112, \"pageref\": \"1\"}, {\"_splash_processing_state\": \"finished\", \"startedDateTime\": \"2020-08-12T02:59:23.394291Z\", \"request\": {\"method\": \"GET\", \"url\": \"https://sp2.baidu.com/-L-Ysjip0QIZ8tyhnq/v.gif?mod=superman%3Acomponents&submod=hotsearch&utype=undefined&superver=supernewplus&portrait=undefined&glogid=3167470754&type=2011&pid=315&isLogin=0&version=PCHome&terminal=PC&qid=3167470909&sid=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498&super_frm=&from_login=&from_reg=&query=&curcard=2&curcardtab=&_r=0.42724906323310996&m=superman%3Acomponents_hotsearchShow&showType=hotword&words=%5B%22\\u4eac\\u6d25\\u5180\\u56db\\u5ddd\\u7b49\\u5730\\u5c40\\u5730\\u6709\\u5927\\u66b4\\u96e8%22%2C%22\\u9ece\\u667a\\u82f1\\u83b7\\u51c6\\u4ee550\\u4e07\\u5143\\u4fdd\\u91ca%22%2C%22\\u5f20\\u7389\\u73af\\u65e0\\u7f6a\\u91ca\\u653e\\u540e\\u529e\\u7406\\u65b0\\u8eab\\u4efd\\u8bc1%22%2C%22\\u62dc\\u767b\\u5ba3\\u5e03\\u54c8\\u91cc\\u65af\\u4f5c\\u4e3a\\u7ade\\u9009\\u642d\\u6863%22%2C%222020\\u5168\\u7403\\u6536\\u5165\\u6700\\u9ad8\\u7537\\u6f14\\u5458%22%2C%22\\u6797\\u5fc3\\u5982\\u970d\\u5efa\\u534e\\u5e86\\u795d\\u7ed3\\u5a5a\\u56db\\u5468\\u5e74%22%5D&pagenum=0\", \"httpVersion\": \"HTTP/1.1\", \"cookies\": [{\"name\": \"BIDUPSID\", \"value\": \"593438C106AB1CB06A8D07285821B5D9\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"PSTM\", \"value\": \"1597201163\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BAIDUID\", \"value\": \"593438C106AB1CB0C0FECD1609F4848C:FG=1\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"H_PS_PSSID\", \"value\": \"32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}], \"queryString\": [{\"name\": \"mod\", \"value\": \"superman%3Acomponents\"}, {\"name\": \"submod\", \"value\": \"hotsearch\"}, {\"name\": \"utype\", \"value\": \"undefined\"}, {\"name\": \"superver\", \"value\": \"supernewplus\"}, {\"name\": \"portrait\", \"value\": \"undefined\"}, {\"name\": \"glogid\", \"value\": \"3167470754\"}, {\"name\": \"type\", \"value\": \"2011\"}, {\"name\": \"pid\", \"value\": \"315\"}, {\"name\": \"isLogin\", \"value\": \"0\"}, {\"name\": \"version\", \"value\": \"PCHome\"}, {\"name\": \"terminal\", \"value\": \"PC\"}, {\"name\": \"qid\", \"value\": \"3167470909\"}, {\"name\": \"sid\", \"value\": \"32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\"}, {\"name\": \"super_frm\", \"value\": \"\"}, {\"name\": \"from_login\", \"value\": \"\"}, {\"name\": \"from_reg\", \"value\": \"\"}, {\"name\": \"query\", \"value\": \"\"}, {\"name\": \"curcard\", \"value\": \"2\"}, {\"name\": \"curcardtab\", \"value\": \"\"}, {\"name\": \"_r\", \"value\": \"0.42724906323310996\"}, {\"name\": \"m\", \"value\": \"superman%3Acomponents_hotsearchShow\"}, {\"name\": \"showType\", \"value\": \"hotword\"}, {\"name\": \"words\", \"value\": \"%5B\\\"\\u4eac\\u6d25\\u5180\\u56db\\u5ddd\\u7b49\\u5730\\u5c40\\u5730\\u6709\\u5927\\u66b4\\u96e8\\\"%2C\\\"\\u9ece\\u667a\\u82f1\\u83b7\\u51c6\\u4ee550\\u4e07\\u5143\\u4fdd\\u91ca\\\"%2C\\\"\\u5f20\\u7389\\u73af\\u65e0\\u7f6a\\u91ca\\u653e\\u540e\\u529e\\u7406\\u65b0\\u8eab\\u4efd\\u8bc1\\\"%2C\\\"\\u62dc\\u767b\\u5ba3\\u5e03\\u54c8\\u91cc\\u65af\\u4f5c\\u4e3a\\u7ade\\u9009\\u642d\\u6863\\\"%2C\\\"2020\\u5168\\u7403\\u6536\\u5165\\u6700\\u9ad8\\u7537\\u6f14\\u5458\\\"%2C\\\"\\u6797\\u5fc3\\u5982\\u970d\\u5efa\\u534e\\u5e86\\u795d\\u7ed3\\u5a5a\\u56db\\u5468\\u5e74\\\"%5D\"}, {\"name\": \"pagenum\", \"value\": \"0\"}], \"headers\": [{\"name\": \"User-Agent\", \"value\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1\"}, {\"name\": \"Referer\", \"value\": \"https://www.baidu.com/\"}, {\"name\": \"Accept\", \"value\": \"*/*\"}, {\"name\": \"Cookie\", \"value\": \"BIDUPSID=593438C106AB1CB06A8D07285821B5D9; PSTM=1597201163; BAIDUID=593438C106AB1CB0C0FECD1609F4848C:FG=1; H_PS_PSSID=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\"}], \"headersSize\": 348, \"bodySize\": -1}, \"response\": {\"bodySize\": 0, \"httpVersion\": \"HTTP/1.1\", \"cookies\": [], \"headers\": [{\"name\": \"Access-Control-Allow-Credentials\", \"value\": \"true\"}, {\"name\": \"Access-Control-Allow-Methods\", \"value\": \"GET, POST, OPTIONS\"}, {\"name\": \"Access-Control-Allow-Origin\", \"value\": \"*\"}, {\"name\": \"Cache-Control\", \"value\": \"no-cache\"}, {\"name\": \"Connection\", \"value\": \"keep-alive\"}, {\"name\": \"Content-Length\", \"value\": \"0\"}, {\"name\": \"Content-Type\", \"value\": \"image/gif\"}, {\"name\": \"Date\", \"value\": \"Wed, 12 Aug 2020 02:59:23 GMT\"}, {\"name\": \"Pragma\", \"value\": \"no-cache\"}, {\"name\": \"Server\", \"value\": \"nginx/1.8.0\"}, {\"name\": \"Tracecode\", \"value\": \"35635184030405974282081210\"}], \"content\": {\"size\": 0, \"mimeType\": \"image/gif\"}, \"headersSize\": 330, \"ok\": true, \"url\": \"https://sp2.baidu.com/-L-Ysjip0QIZ8tyhnq/v.gif?mod=superman%3Acomponents&submod=hotsearch&utype=undefined&superver=supernewplus&portrait=undefined&glogid=3167470754&type=2011&pid=315&isLogin=0&version=PCHome&terminal=PC&qid=3167470909&sid=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498&super_frm=&from_login=&from_reg=&query=&curcard=2&curcardtab=&_r=0.42724906323310996&m=superman%3Acomponents_hotsearchShow&showType=hotword&words=%5B%22\\u4eac\\u6d25\\u5180\\u56db\\u5ddd\\u7b49\\u5730\\u5c40\\u5730\\u6709\\u5927\\u66b4\\u96e8%22%2C%22\\u9ece\\u667a\\u82f1\\u83b7\\u51c6\\u4ee550\\u4e07\\u5143\\u4fdd\\u91ca%22%2C%22\\u5f20\\u7389\\u73af\\u65e0\\u7f6a\\u91ca\\u653e\\u540e\\u529e\\u7406\\u65b0\\u8eab\\u4efd\\u8bc1%22%2C%22\\u62dc\\u767b\\u5ba3\\u5e03\\u54c8\\u91cc\\u65af\\u4f5c\\u4e3a\\u7ade\\u9009\\u642d\\u6863%22%2C%222020\\u5168\\u7403\\u6536\\u5165\\u6700\\u9ad8\\u7537\\u6f14\\u5458%22%2C%22\\u6797\\u5fc3\\u5982\\u970d\\u5efa\\u534e\\u5e86\\u795d\\u7ed3\\u5a5a\\u56db\\u5468\\u5e74%22%5D&pagenum=0\", \"status\": 200, \"statusText\": \"OK\", \"redirectURL\": \"\"}, \"cache\": {}, \"timings\": {\"blocked\": -1, \"dns\": -1, \"connect\": -1, \"ssl\": -1, \"send\": 0, \"wait\": 107, \"receive\": 0}, \"time\": 107, \"pageref\": \"1\"}, {\"_splash_processing_state\": \"finished\", \"startedDateTime\": \"2020-08-12T02:59:23.422817Z\", \"request\": {\"method\": \"GET\", \"url\": \"https://www.baidu.com/sugrec?prod=pc_his&from=pc_web&json=1&sid=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498&hisdata=&req=2&csor=0\", \"httpVersion\": \"HTTP/1.1\", \"cookies\": [{\"name\": \"BIDUPSID\", \"value\": \"593438C106AB1CB06A8D07285821B5D9\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"PSTM\", \"value\": \"1597201163\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BAIDUID\", \"value\": \"593438C106AB1CB0C0FECD1609F4848C:FG=1\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BD_HOME\", \"value\": \"1\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"H_PS_PSSID\", \"value\": \"32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BD_UPN\", \"value\": \"143354\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}], \"queryString\": [{\"name\": \"prod\", \"value\": \"pc_his\"}, {\"name\": \"from\", \"value\": \"pc_web\"}, {\"name\": \"json\", \"value\": \"1\"}, {\"name\": \"sid\", \"value\": \"32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\"}, {\"name\": \"hisdata\", \"value\": \"\"}, {\"name\": \"req\", \"value\": \"2\"}, {\"name\": \"csor\", \"value\": \"0\"}], \"headers\": [{\"name\": \"User-Agent\", \"value\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1\"}, {\"name\": \"Accept\", \"value\": \"application/json, text/javascript, */*; q=0.01\"}, {\"name\": \"Referer\", \"value\": \"https://www.baidu.com/\"}, {\"name\": \"Cookie\", \"value\": \"BIDUPSID=593438C106AB1CB06A8D07285821B5D9; PSTM=1597201163; BAIDUID=593438C106AB1CB0C0FECD1609F4848C:FG=1; BD_HOME=1; H_PS_PSSID=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498; BD_UPN=143354\"}], \"headersSize\": 417, \"bodySize\": -1}, \"response\": {\"bodySize\": 53, \"httpVersion\": \"HTTP/1.1\", \"cookies\": [], \"headers\": [{\"name\": \"Content-Length\", \"value\": \"53\"}, {\"name\": \"Content-Type\", \"value\": \"text/plain; charset=UTF-8\"}, {\"name\": \"Date\", \"value\": \"Wed, 12 Aug 2020 02:59:23 GMT\"}], \"content\": {\"size\": 53, \"mimeType\": \"text/plain; charset=UTF-8\"}, \"headersSize\": 98, \"ok\": true, \"url\": \"https://www.baidu.com/sugrec?prod=pc_his&from=pc_web&json=1&sid=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498&hisdata=&req=2&csor=0\", \"status\": 200, \"statusText\": \"OK\", \"redirectURL\": \"\"}, \"cache\": {}, \"timings\": {\"blocked\": -1, \"dns\": -1, \"connect\": -1, \"ssl\": -1, \"send\": 1, \"wait\": 23, \"receive\": 0}, \"time\": 24, \"pageref\": \"1\"}, {\"_splash_processing_state\": \"finished\", \"startedDateTime\": \"2020-08-12T02:59:23.503417Z\", \"request\": {\"method\": \"GET\", \"url\": \"https://www.baidu.com/cache/fpid/chromelib_1_1.js?_=1597201163314\", \"httpVersion\": \"HTTP/1.1\", \"cookies\": [{\"name\": \"BIDUPSID\", \"value\": \"593438C106AB1CB06A8D07285821B5D9\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"PSTM\", \"value\": \"1597201163\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BAIDUID\", \"value\": \"593438C106AB1CB0C0FECD1609F4848C:FG=1\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BD_HOME\", \"value\": \"1\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"H_PS_PSSID\", \"value\": \"32292_1438_31671_32536_32327_32349_32046_32393_32116_32498\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}, {\"name\": \"BD_UPN\", \"value\": \"143354\", \"path\": \"\", \"domain\": \"\", \"httpOnly\": false, \"secure\": false}], \"queryString\": [{\"name\": \"_\", \"value\": \"1597201163314\"}], \"headers\": [{\"name\": \"User-Agent\", \"value\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1\"}, {\"name\": \"Accept\", \"value\": \"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01\"}, {\"name\": \"Referer\", \"value\": \"https://www.baidu.com/\"}, {\"name\": \"X-Requested-With\", \"value\": \"XMLHttpRequest\"}, {\"name\": \"Cookie\", \"value\": \"BIDUPSID=593438C106AB1CB06A8D07285821B5D9; PSTM=1597201163; BAIDUID=593438C106AB1CB0C0FECD1609F4848C:FG=1; BD_HOME=1; H_PS_PSSID=32292_1438_31671_32536_32327_32349_32046_32393_32116_32498; BD_UPN=143354\"}], \"headersSize\": 507, \"bodySize\": -1}, \"response\": {\"bodySize\": 0, \"httpVersion\": \"HTTP/1.1\", \"cookies\": [], \"headers\": [], \"content\": {\"size\": 0, \"mimeType\": \"\"}, \"headersSize\": 0, \"ok\": false, \"url\": \"https://www.baidu.com/cache/fpid/chromelib_1_1.js?_=1597201163314\", \"status\": 0, \"statusText\": \"cancelled\", \"redirectURL\": \"\"}, \"cache\": {}, \"timings\": {\"blocked\": -1, \"dns\": -1, \"connect\": -1, \"ssl\": -1, \"send\": 0, \"wait\": 0, \"receive\": 3}, \"time\": 3, \"pageref\": \"1\"}], \"pages\": [{\"id\": \"1\", \"title\": \"\\u767e\\u5ea6\\u4e00\\u4e0b\\uff0c\\u4f60\\u5c31\\u77e5\\u9053\", \"startedDateTime\": \"2020-08-12T02:59:22.951998Z\", \"pageTimings\": {\"onContentLoad\": 420, \"onLoad\": 554, \"_onStarted\": 0, \"_onPrepareStart\": 555}}]}}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "url='http://192.168.99.102:8050/render.har?url=https://www.baidu.com'\n",
    "response=requests.get(url)\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回json格式数据，其中会有显示这些HAR数据\n",
    "\n",
    "### 5.render.json\n",
    "\n",
    "此接口包含了前面接口的所有功能，返回结果是JSON格式.\n",
    "curl http://localhost:8050/render.json?url=https://httpbin.org&html=1&har=1\n",
    "\n",
    "表示返回网页源代码和har数据\n",
    "\n",
    "### execute\n",
    "\n",
    "此接口才是最为强大的接口。execute接口可以实现一些交互操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from urllib.parse import quote\n",
    "lua=\"\"\"\n",
    "function main(splash)\n",
    "    return 'hello'\n",
    "end\n",
    "\"\"\"\n",
    "\n",
    "url='http://192.168.99.102:8050/execute?lua_source='+quote(lua)\n",
    "response=requests.get(url)\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们用Python中的三引号将Lua脚本包括起来，然后用urllib.parse模板里的quote()方法将脚本进行URL转码，随后构造了Splash请求URL，将其作为lua_source参数传递\n",
    "\n",
    "我们所说的Lua脚本均可以用此方法与Python进行对接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"status\": 200, \"html\": \"{\\n  \\\"args\\\": {}, \\n  \\\"headers\\\": {\\n    \\\"Accept-Encoding\\\": \\\"gzip, deflate\\\", \\n    \\\"Accept-Language\\\": \\\"en,*\\\", \\n    \\\"Host\\\": \\\"httpbin.org\\\", \\n    \\\"User-Agent\\\": \\\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1\\\", \\n    \\\"X-Amzn-Trace-Id\\\": \\\"Root=1-5f335f06-339f299a12dd6ab6bca1e590\\\"\\n  }, \\n  \\\"origin\\\": \\\"39.171.13.220\\\", \\n  \\\"url\\\": \\\"http://httpbin.org/get\\\"\\n}\\n\", \"url\": \"http://httpbin.org/get\"}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from urllib.parse import quote\n",
    "lua=\"\"\"\n",
    "function main(splash,args)\n",
    "    local treat=require(\"treat\")\n",
    "    local response=splash:http_get(\"http://httpbin.org/get\")\n",
    "    return {\n",
    "         html=treat.as_string(response.body),\n",
    "         url=response.url,\n",
    "         status=response.status\n",
    "         }\n",
    "end\n",
    "\"\"\"\n",
    "url='http://192.168.99.102:8050/execute?lua_source='+quote(lua)\n",
    "response=requests.get(url)\n",
    "print(response.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splash负载平衡 专题\n",
    "\n",
    "Splash做页面抓取时，如果爬取的量非常大，任务非常多，用 Splash 服务来处理的话，\n",
    "可能压力太大了，此时可以考虑搭建一个负载均衡器来把压力分散到各个服务器上，这相当于多台机器\n",
    "多个服务共同参与任务的处理，可以减小单个Splash服务的压力\n",
    "\n",
    "### 1. 配置Splash服务\n",
    "\n",
    "要搭建Splash负载均衡，首先要有多个Splash服务。\n",
    "\n",
    "假如这里在4台远程主机的 8050 端口上都\n",
    "开启了Splash服务，它们的服务地址分别为\n",
    "41.159.27.223:8050  41.159.27.221:8050  41.159.27.9:8050\n",
    "41.159.117.119:8050\n",
    "\n",
    "这4个服务完全一致，都是通过Docker Splash镜像开启的 访问其中任何1个服务时，\n",
    "都可以使用Splash服务\n",
    "\n",
    "### 2.配置负载均衡\n",
    "\n",
    "接下来，可以选用任意一台带有公网 IP 的主机来配置负载均衡。\n",
    "\n",
    "首先，在这台主机上装好Nginx,然后修改Nginx的配置文件nginx.conf"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "http {\n",
    "    upstream splash {\n",
    "        least_conn;\n",
    "        server 41.159.27.223:8050;\n",
    "        server 41.159.27.221:8050;\n",
    "        server 41.159.27.9:8050;\n",
    "        server 41.159.117.119:8050;\n",
    "}\n",
    "server {\n",
    "    listen 8050;\n",
    "    location / {\n",
    "    proxy_pass http://splash;\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这样我们通过叩stream字段定义了个名字叫作splash的服务集群配置.\n",
    "\n",
    "其中 least_conn代表最少链接负载均衡，它适合处理请求处理时间长短不一造成服务器过载的情况\n",
    "\n",
    "也可以用下面的方法实现配置："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "upstream splash {\n",
    "server 41.159.27.223:8050;\n",
    "server 41.159.27.221:8050;\n",
    "server 41.159.27.9:8050;\n",
    "server 41.159.117.119:8050;\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "这样默认以轮询策略实现负载均衡，每个服务器的压力相同\n",
    "\n",
    "此策略适合服务器配置相当、无状态且短平快的服务使用\n",
    "\n",
    "另外，我们还可以指定权重，配置如下"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "upstream splash {\n",
    "server 41.159.27.223:8050 weight=4;\n",
    "server 41.159.27.221:8050 weight=2;\n",
    "server 41.159.27.9:8050 weight=2;\n",
    "server 41.159.117.119:8050 weight=1;\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "指定各个服务的权重，权重越高，分配到处理的请求越多。假如不同的服务器配\n",
    "别比较大的话，可以使用此种配置\n",
    "\n",
    "最后，还有一种IP散列负载均衡，配置如下："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "upstream splash {\n",
    "ip_hash;\n",
    "server 41.159.27.223:8050;\n",
    "server 41.159.27.221:8050;\n",
    "server 41.159.27.9:8050;\n",
    "server 41.159.117.119:8050;\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "服务器根据请求客户端的 IP 地址进行散列计算，确保使用同一个服务器响应请求，\n",
    "这种策略适合有状态的服务，比如用户登录后访问某个页面的情形 对于Splash来说，\n",
    "不需要应用此设置\n",
    "\n",
    "我们可以根据不同的情形选用不同的配置，配置完成后重启一下 Nginx 服务：\n",
    "sudo nginx -s reload\n",
    "\n",
    "### 3.配置认证\n",
    "\n",
    "现在 Sp ash 是可以公开访问的，如果不想让其公开访问，还可以配置认证，\n",
    "这仍然借助于Nginx\n",
    "可以在 server的location 字段中添加 auth_basic auth_basic_user_file 字段"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "http {\n",
    "upstream splash {\n",
    "    least_conn;\n",
    "    server 41.159.27.223:8050;\n",
    "    server 41.159.27.221:8050;\n",
    "    server 41.159.27.9:8050;\n",
    "    server 41.159.117.119:8050;\n",
    "}\n",
    "server {\n",
    "    listen 8050;\n",
    "    location / {\n",
    "        proxy_pass http://splash;\n",
    "        auth_basic \"Restricted\";\n",
    "        auth_basic_user_file /etc/nginx/conf.d/.htpasswd;\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里使用的用户名和密码配置放置在etc/nginx/conf.d 目录下，我们需要使用htpasswd命令创建\n",
    "例如，创建一个用户名为 admin 的文件"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "htpasswd -c .htpasswd admin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来就会提示我们输入密码，输入两次之后，就会生成密码文件，\n",
    "\n",
    "cat .htpasswd\n",
    "\n",
    "admin: 5ZBxQrOrCqwbc\n",
    "\n",
    "配置完成后，重启 Nginx 服务就配置完毕："
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "sudo nginx -s reload"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.测试\n",
    "\n",
    "最后，我们可以用代码来测试下负载均衡的配置，看看到底是不是每次请求切换IP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from urllib.parse import quote\n",
    "import re\n",
    "lua ='''\n",
    "function main(splash, args)\n",
    "local treat = require(\"treat\")\n",
    "local response = splash:http_get(\"http://httpbin. org/get\")\n",
    "return treat.as_string(response.body)\n",
    "end\n",
    "'''\n",
    "\n",
    "url = 'http://splash:8050/execute?lua_source=' + quote(lua)\n",
    "response = requests.get(url, auth=('admin','admin'))\n",
    "ip = re.search('(\\d+\\.\\d+\\.\\d+\\.\\d+)', response.text) .group(1)\n",
    "print(ip)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里URL中的splash字符串请自行替换成自己的Nginx服务器IP。\n",
    "这里我修改了Hosts，设置了splash为Nginx服务器IP。\n",
    "\n",
    "多次运行代码之后，可以发现每次请求的IP都会变化，\n",
    "说明负载均衡已经成功实现.\n",
    "\n",
    "本节中，我们成功实现了负载均衡的配置\n",
    "\n",
    "配置负载均衡后，可以多个 Splash 服务共同合作，\n",
    "轻单个服务的负载，这还是比较有用的"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PyCharm (PycharmWorkspace)",
   "language": "python",
   "name": "pycharm-7b63a738"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
